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Abstract 

The need for accountability in school psychological services has received 
increased attention. School psychologists must be able to demonstrate the effectiveness of 
their services. Data collection procedures can improve services to clients and systems, 
enhance the delivery of a system’s psychological services program, validate the need for 
psychological services in schools, and promote system change and reform initiatives. 

This paper provides an overview of school psychology accountability strategies. 
The importance and rationale of incorporating accountability strategies into practice is 
discussed. Enumerative, process, and outcome data are reviewed. Practical data 
collection strategies are illustrated through examples and case studies. A systematic 
problem-solving framework is presented that can be used for making data-based 
educational decisions. 
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Rationale 

Accountability has been an integral part of educational reform initiatives. The 
pubic demand for accountability in education has led to increased efforts to demonstrate 
the effectiveness of school psychological services. School psychologists must be able to 
provide evidence that their services result in favorable outcomes for clients. In School 
Psychology: A Blueprint for Training and Practice II (1997), Ysseldyke et al. have 
designated data-based decision making and accountability as a school psychology 
leadership and function domain. New assessment and decision making practices have 
resulted in an increase in accountability practices among practitioners and training 
programs (Fairchild, 1985). The changes in school psychologists’ roles from the 
traditional special education “test-place” role to data-based problem-solving has also 
contributed to an increased emphasis in data collection and accountability. 

Surveys indicate that approximately 60% of school psychologists collect some 
type of accountability data (Fairchild & Zins, 1992; Zins & Fairchild 1986). A 
comparison of 1986 and 1992 accountability data suggested that the percentage of school 
psychologists involved in accountability efforts has not changed significantly over time. 
Unfamiliarity with accountability procedures was reported by survey respondents as the 
most significant barrier to collecting accountability data followed by the amount of time 
required to collect data. Most respondents (57%) collected data because it was required 
by their supervisor. Others collected data out of personal choice (52.7%), to plan for 
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future service delivery (52.7%) or because it was required by their central office (40%) or 
state department of education (33%). 

Zins (1997) discussed the need for accountability in school psychology. One of 
the main reasons for incorporating accountability into practice is for the benefit of clients 
and consumers of school psychological services. Consumer satisfaction information can 
provide valuable information on client/system needs.. Accountability data can be 
collected to determine consumer satisfaction with services. School psychologists also 
can collect data on the effectiveness of the services they provide to clients. By assessing 
outcomes, psychologists can determine the impact of their services and use information 
to improve the services that they offer to clients. 

A second reason for collecting accountability data is to assist practitioners with 
their professional development. Accountability data can be used to evaluate professional 
goals and skills. School psychologists can collect information on the types of services 
they provide to clients and outcomes. With this information, psychologists can make 
data-based decisions on whether the time they spent involved in various activities is 
resulting in positive outcomes for students. For example, school psychologists can 
evaluate the time spent on conducting formal evaluations and determine if they lead to 
favorable outcomes for clients. If one notices that too much time is being spent on 
assessment activities that do not result in improved academic performance, psychologists 
may be able to make the needed changes in the services they provide to better meet the 
needs of clients. 
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Accountability practices also can lead to improvement in a system’s overall 
psychological services program. System-wide implementation of accountability practices 
can provide useful information on the strengths, weakness, and needs of a school system. 
Accountability data may provide system administrators with information on referral 
patterns, groups of consumers receiving services, and time spent in classification 
activities. Patterns in the data may suggest areas in need of change on a systems level. 
Accountability efforts also offer potential benefits to the overall profession of school 
psychology. Data can be used to support the need for decreasing student-psychologist 
ratios, support changes in school psychologists’ roles, enhance public relations, and 
increase the community’s awareness of the services that school psychologists can offer. 

Definition of Accountability 

Several different definitions of accountability have been offered in the psychology 
and education literature. Lessinger (1970) defined accountability has an “agreement by 
school staff members to contract on the basis of providing a service answerable to 
stipulated performance standards” (p. 318). Zins (1984) proposed the following 
definition of accountability: 

Accountability is an evaluative effort designed to systematically gather 
information relevant to the performance of school psychologists. It enables them 
to demonstrate the effectiveness of their services to others and it provides an 
evaluation of how well they have met their performance objectives. It is 
concerned with both quantitative and qualitative aspects of practice, and ... it is 
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particularly useful in improving service delivery and enhancing professional 
development (p. 58). 

Trachtman (1981) made a distinction between accountability-imposed and 
accountability-offered. Accountability-imposed refers to documenting to justify services 
rendered in response to a request from a higher authority. This may happen when a 
supervisor requires a school psychologist to document hours involved in various 
activities or the number of evaluations completed. Accountability-offered involves the 
initiation of accountability procedures out of professional responsibility. School 
psychologists may initiate data collection to evaluate their effectiveness, improve 
perceptions of consumers, and enhance service delivery. 

Fairchild and Zins (1992) found that majority of school psychologists collect 
accountability data as a result of a demand from a supervisor (accountability-imposed). 
Although both types of accountability may lead to improvements in service delivery, 
there is a greater chance that school psychologists will change their professional behavior 
when accountability procedures are self-initiated (Zins, 1984). Furthermore, when 
accountability procedures are imposed by authority figures, school psychologists may 
have minimal input into how their services are evaluated or the criteria by which their 
performance is judged. Imposed accountability strategies such as collecting data on the 
number of evaluations completed provides a limited view of the role of school 
psychologists and does not provide information that would necessarily support role 
change. In cases where accountability is imposed, it is critical that school psychologists 
collect data beyond what is minimally required. School psychologists should gather 
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additional information that can be used to evaluate the effectiveness of interventions and 
support a qualitative and quantitative expansion of psychological services. 

Types of Accountability Data 

Zins (1995) identified three types of accountability data: enumerative, process, 
and outcome. Enumerative data refers to information that is gathered by tallying the 
number of various activities occurred. For example, a school psychologist may tally the 
number of evaluations completed within a year, the number of students that received 
direct psychological intervention, or the number of teacher referrals. School 
psychologists also may record the number of hours spent in various activities and 
calculate the percentage of time engaged in such activities as consultation, counseling, 
assessment, and professional development. 

Of those school psychologists who collect accountability data, enumerative data is 
gathered most often. In the survey conducted by Fairchild and Zins (1992), 97% of those 
school psychologists who reported that they obtain accountability data indicated that they 
gathered enumerative data. Although the relationship between the type of accountability 
data collected and the reason for collecting data was not examined by the researchers, it is 
likely that many of the school psychologists who collect enumerative data do so out of a 
request from a supervisor. School psychologists are often required to maintain records of 
the number of evaluations completed and hours spent in various types of activities. 

Process data are collected to evaluate the implementation of services or assess 
how the services are perceived by clients (i.e., consumer satisfaction). This type of data 
provides information on how things were done or how something was experienced rather 
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than the outcome of the service. School psychologists also may assess client and system 
satisfaction with services rendered. 

Fairchild and Zins (1992) found that 36% of school psychologists who utilize 
accountability procedures collect process data. Process data are relatively easy to collect 
through surveys and interviews. Although there are many limitations to attitudinal 
measures, process data collection procedures can produce useful and valid information 
about how clients perceive psychological services and the implementation of those 
services. 

Outcome data describe the changes in client behavior (child, parent, and system) 
that result from a service. School psychologists collect outcome data to measure goal 
attainment or changes in targeted behavior due to specific interventions. Although 
process data often yields the most useful information, few school psychologists collect 
accountability data. In their survey of school psychologists (1992), Fairchild and Zins 
found that 26% of those involved in accountability practices collect outcome data. 

School psychologist may not obtain outcome data due to the inherent difficulty in 
collecting this type of information. 

Accountability Procedures 

Procedures for collecting accountability data often involve tallying the hours 
performed in various activities. The daily log is often used to record time and the types 
of services performed (Figure 1). Each specific activity is then coded under a general 
category such as Assessment, Counseling, Professional Development, Supervision, and 
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FIGURE 1 

School Psychologist Daily Log 



Name: District: 

Week: School: 

Activity Codes: A = Assessment, CON = Consultation, C = Counseling, 
PD = Professional Development, S = Supervision, IT = Inservice Training 



Day 


Activity 


Code 


Hours 


Monday 








Tuesday 








Wednesday 








Thursday 








Friday 








Weekly 

Total 


■'y • ■ • •• y if. 




A = 
CON = 
C = 

PD = 

S = 

IT = 


Cumm. 

Hours 


: I 

» v: i 

ir ,**y - < 1 

!• • .. .f-f ; 

kill ■ '■ . %?; 


- •' 


A = 
CON = 
C = 

PD = 

S = 

IT = 
Total = 



0 

ERIC 






10 



Accountability Strategies 10 



Inservice Training. Classification systems can contain more specific categories such as 
the activity categories suggested by Fairchild (1975). However, too many categories can 
be confusing and difficult to interpret. The percentage of time spent in activities can be 
calculated by dividing the number of hours spent in the activity category by the total 
number of hours. Monthly, semester, and annual reports can be generated and displayed 
graphically (Figure 2). 

School psychologists also can collect data on the number of teacher referrals, 
students served, and evaluations conducted. Referrals patterns can be analyzed by 
differentiating the types of referrals made (behavioral, academic, or combined) and the 
types of interventions that resulted from the referral (direct intervention, consultation, 
counseling, parent training, etc.). The amount of time elapsed between the referral and 
implementation of the intervention also is useful to examine. Immediacy of feedback can 
be used to justify new personnel and is often valued by consumers and system 
administrators (Fairchild, 1975). Groups of students served (special education versus 
general education) can be easily determined and used to identify target groups and 
underserved populations. 

School psychologists have often been evaluated based upon the number of 
assessments completed. This is a limited view of the functioning of school psychologists. 
The traditional “test-place” role has been criticized because it does not always lead to 
effective interventions for students (Reschly, 1988). In fact, school psychologists can 
demonstrate through enumerative data that spending the majority of their time in 
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FIGURE 2 

Percentage of Time Spent in Various School Psychology Activities 



Inservice Training 
4% 




34% 



classification activities is not serving the needs of their clients. The number of 
evaluations completed and time spent in classification activities may be used to justify 
increasing school psychologist-student ratios and to advocate for the broadening of 
school psychologists’ roles. 

Process data can be collected by surveying teachers, parents, and students to 
assess satisfaction of services rendered. Fairchild (1985) described a consumer feedback 
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system that utilizes a series of questionnaires, a record-keeping procedure, and an 
interview format to elicit information regarding school psychology intern effectiveness. 
This system can be applied to other school psychology practitioners. Questionnaires are 
distributed to teachers and parents to evaluate satisfaction with assessment and 
consultation services. Student questionnaires are completed to evaluate student 
satisfaction with counseling services. Respondents use a Likert rating scale (1 - 5) to 
respond to each item. Mean ratings per item are calculated along with the return rates. 
Strengths, weakness, and recommendations are generated from each of the 
questionnaires. On-going record keeping allows trainers or supervisors to monitor 
consumer feedback on a regular basis, identify areas that require further 
training/experience, and enhance public relations and practitioner visibility. 

Surveys also may be used to assess consumer satisfaction with particular 
programs. Figure 3 is a survey developed to evaluate teacher satisfaction with Kid Talk 
Teams. Kid Talk Teams are grade-level teams composed of teachers, student services 
staff (psychologists, counselors, pupil personnel workers), and administrators that use a 
systematic problem-solving process to develop prereferral interventions (Rosenfield & 
Gravois, 1996). This survey was designed to measure teacher satisfaction and to evaluate 
the degree to which teams were following the process. Items such as “problems were 
defined in concrete and measurable terms” were developed to determine the extent to 
which the teams followed the systematic problem-solving process. 

The implementation of the problem-solving process by Kid Talk Teams also was 
assessed through systematic observations of the teams. Bartels and Mortenson (1999) 
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FIGURE 3 

Kid Talk Team Process Survey 



a. Your position this year: 

teacher support staff 

b. Kid Talk Team(s) you participated in this year: 

grade six grade seven grade eight related arts 

Please rate each of the following statements by circling the number 

1 = STRONGLY DISAGREE 2 = DISAGREE 3 = SOMETIMES AGREE 
4 = AGREE 5 = STRONGLY AGREE 

1. 1 made a significant contribution to the Kid Talk process. 1 2 3 4 5 

2. 1 am invested in the success of the Kid Talk process. 1 2 3 4 5 

3. Kid Talk was successful in developing effective interventions 

for students. 1 2 3 4 5 

4. The interventions that the Kid Talk team selected were supported 

by the members. 1 2 3 4 5 

5. The Kid Talk team defined problems in concrete and measurable 

terms. 1 2 3 4 5 

6. The Kid Talk team resolved differences in opinion regarding 

interventions. 1 2 3 4 5 

7. The interventions developed the Kid Talk team were based on 

the needs of children. 1 2 3 4 5 

8. Organizational issues limited the Kid Talk team’s ability to do 

what was in the best interest of the child. 1 2 3 4 5 



9. Parent input was used by the Kid Talk Team to develop 
interventions. 



1 2 3 4 5 
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FIGURE 3 - CONTINUED 

10. The Kid Talk team listened to my concerns. 1 2 3 4 5 

11. The Kid Talk team used time efficiently. 1 2 3 4 5 

12. Parent input was used by the Kid Talk team to identify 

problems. 1 2 3 4 5 

13. The Kid Talk team had a sufficient amount of time to 

complete the work. 1 2 3 4 5 

14. The Kid Talk team was able to find a meeting time that was 1 2 3 4 5 

convenient for all members. 

15. The Kid Talk team focused on creating a match between the 

student and the curriculum/classroom environment. 1 2 3 4 5 



16. Kid Talk members received sufficient training. 



1 2 3 4 5 



17. Within our building, there were adequate resources for 

the implementation of the Kid Talk team’s recommendations. 1 2 3 4 5 

18. The Kid Talk team had sufficient knowledge to design and 

implement interventions. 1 2 3 4 5 

19. The Kid Talk team used collaborative problem solving to 

improve student learning. 1 2 3 4 5 

20. The Kid Talk team collected and used data to identify goals 

for students and develop, monitor, and evaluate interventions. 1 2 3 4 5 

21. The Kid Talk team focused on variables that teachers could 

control or change. 1 2 3 4 5 
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developed a Process Observation Form to assess Kid Talk Teams’ use of systematic 
problem-solving (Figure 4). Survey and observational data were summarized and shared 
with a school improvement team to identify program strengths and weakness, establish 
goals for the project, and determine training needs for team members. 

The effectiveness of interventions is often collected through the use of single- 
subject designs. Due to ethical concerns regarding the removal of needed interventions, 
advanced single-subject designs such as A-B-A and A-B-A-B are not recommended. 
These designs often involve removing the intervention for a period of time to determine if 
the change in behavior can be attributed to the intervention or other factors such as 
maturation. 

Simple single-subject designs typically involve several measurements of the 
targeted behavior before the intervention is implemented (baseline). Following 
intervention implementation, the occurrence of the target behavior is regularly measured. 
When significant increases in the targeted behavior across conditions are observed and 
other factors remain constant, changes in the targeted behavior are attributed to the 
success of the intervention. 

Goal Attainment Scaling (GAS) is an outcome-based procedure that assesses 
student performance toward observable and quantitatively defined goals. Mather and 
Barbrack (1984) developed a method of measuring counseling goals using GAS. Goals 
are mutually determined and ranked according to priority by the counselor and student. 
Each goal is broken down into specific observable and quantifiable behaviors and is 
assigned weights (+2, +1, 0, -1, -2) that represent long-term, intermediate, and short-term 
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FIGURE 4 

Process Observation Form 

Date _ 

Grade 



Student’s first name School 

Observer Psychologist 



Referral Concern “ 



□ 



Behavior 



D Academic D Combined 



Case # 



□ 1 . 

o 

□ 2 . 

□ 3. 

o 

□ 4. 

□ 5. 

□ 6 . 

o 

o 

o 

o 



Meeting is facilitated by SST representative 
Psychologist was present 

Minutes are taken on appropriate form 

Problem-solving conducted only on students listed as “initial refenab” 
non-agenda students were discussed 

Referral concern described in observable and measurable terms 
Instructional issues are addressed 

Baseline data is presented or plan to collect baseline data is developed 
CBA: Accuracy and fluency 
Percent work completed 

Test scores: Standardized and classroom-based 
Systematic classroom observation 



□ 7. 

□ 8 . 

□ 9. 

□ 10 . 
'□ ll. 

□ 12 . 

□ 13. 

□ 14. 

□ 15. 



Antecedents (e.g., setting, task, time of day) and consequences are 
discussed 

An observable and measurable goal has been set 

Teacher(s) are actively involved in planning the intervention 

Specifics of intervention are outlined 

Who will implement the plan is specified 

What will be implemented is specified 

Wtan the intervention will be implemented is specified 

Facilitator/other team member ascertains that all team members are in agreement 
regarding u>ho will implement uJirtt by u'/ien 

Follow-up date is set for next Kid talk meeting 



Total Percent completed steps : 
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goals. During the counseling sessions, the counselor provides feedback to the client 
regarding performance. Figure 5 is an example of a Goal Attainment Scale form. 

It is important to consider how outcome data will be collected before the 
intervention is implemented. An evaluation plan should be developed to assess student 
response to interventions. Often, outcome data can be incorporated as part of the 
intervention. Figure 6 is an example of an intervention developed to increase student on- 
task behavior. The teacher and student indicate on the chart the percentage of each task 
completed. Data from the charts can be collected and analyzed to determine the student’s 
response to the intervention by comparing the student’s current performance with 
baseline data. Students and teachers also receive feedback on whether the student has 
reached the goal at the end of each task, class, and day. 

One of the most efficient and effective methods of collecting outcome data on 
student response to academic interventions is through the use of curriculum-based 
measurement (CBM) procedures (Shinn, 1989). CBM procedures are quick and easy to 
administer. Studies have found that CBM procedures are valid, reliable, and culturally 
fair measures of academic achievement (Fuchs & Fuchs, 1992; Marston, 1989; Shinn, 
Good, Knutson, Tilly, & Collins, 1992). CBM procedures are relatively sensitive to 
academic improvement over time and can be used to monitor academic interventions 
(Fuchs, Fuchs, & Hamlett, 1989). Students can set their own goals using baseline data 
and chart their progress over time. 
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FIGURE 5 



Instructional Consultation Student Documentation Form 

Student’s Name Grade Dateofflirth Date Started 

Teacher’s Name * Case Manager School 



Goal Attainment Scale (GAS) 



Step 1: 

Initial descnption of concern 










Step 2: 

Prioritize 


Importance 12 3 4 
(student at instructional 
level! Y N) 


Importance 12 3 4 
(student it instructional 
level? Y N) 


Importance 12 3 4 

(student at instructional 
level? Y N) 


Importance 12 3 4 

(student it instructional 
level? Y N) 


Step 3: 

Observablc/measurablc 
statement of current perfor- 
mance (following baseline) 


Date collected 


Date collected 


Date collected 


Date collected 










Step 4: 

Short term goal; 

Expected performance 
in weeb (4-dweeb) 


Date consistently 
attained 


Date consistently 
attained 


Date consistently 
attained 


Dare consistently 
attained 


Step 5: 
interim goal: 
Expected behavior 
in weeks 


Date consistently 
attained 


Date consistently 
attained 


Date consistendy • 
attained 


Date consistently 
attained 


Step 6: 

Long-term goal: 
Expected behavior 
in weeb 


Date consistently 
attained 


Dare consistently 
attained 


Date consistently 
attained 


Date consistently 
attained 












19 



Accountability Strategies 19 



FIGURE 6 



APPROACH TO TASK CHART 



DATE SUBJECT TASK 



BEFORE 



AFTER 



WHAT TO DO 


STUDENT: 

Think about 
? 


TEACHER: 

Check 

V 




STUDENT: 

Check 

V 


TEACHER: 

Check 

V 


MATERIALS 


What materials do 
I need? 


Identified 

materials 

Yes No 




Did I use 
materials 
correctly? 

Yes No 


Used materials 
correctly 
Yes No 


DIRECTIONS 


How do I 
complete the 
task? 


Explained 
directions 
Yes No 




Did I follow 
directions? 
Yes No 


Followed 
directions 
Yes No 


TIME 


What are the time 
limits? 


Identified 
time limits 

Yes No 




Did I complete 
75% of the 
assignment? 
Yes No 


Percentage of 

assignment 

completed 

% 
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A Problem-Solving Framework for Accountability Procedures 

Zins (1984) developed a behavioral problem-solving framework to be used with 
accountability procedures. It is based on a scientific method of defining problems, 
operationalizing information, and evaluating results. The first step in the model involves 
problem identification and formulation. The school psychologist must first determine 
who needs or wants the accountability data and the purpose of collecting the information. 
Job descriptions and discussions with administrators can help prioritize and target areas 
where data will be collected. School psychologists must begin with identifying areas for 
improvement and skill development. 

After specific areas have been targeted, the next step is to determine possible 
sources of information and various ways of gathering the data. Sources of information 
may include teachers, parents, students, and supervisors. A multisource and multimethod 
approach is recommended to increase the validity and reliability of the results. 

The next step in the process is to determine the most appropriate data collection 
instruments. These procedures have been described in previous sections of this paper. It 
is important to consider such factors as time, ease of administration, and availability of 
data. Enumerative, process, and outcome data collection procedures can be used 
depending upon the questions being asked. 

Once the plan has been developed, data is then collected, interpreted, and 
analyzed. It is important to periodically analyze and review data. Progress toward goals 
can be evaluated periodically rather than waiting until the end of the year. As patterns 
emerge, needed changes in service delivery can be made. 
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Accountability data is then disseminated to consumers and supervisors. 
Information must be shared with major stakeholders to provide evidence of the 
effectiveness of services. Zins (1984) suggested the construction of an “accountability 
report” that summarizes the data. Technology can be used to present the data graphically 
so it is easy to interpret. 

After data have been shared, it is important that school psychologists reflect on 
the accountability process. To what extent did the procedures used provide useful 
information? Did the accountability data lead to improved performance? Did the 
strategies used provide reliable and valid data? What are the implications for 
professional development/further training? 




Guidelines for Developing Accountability Procedures 
The problem-solving framework proposed by Zins (1984) can help school 
psychologists plan and develop relevant, efficient, and useful accountability procedures. 
Data collected should be related to areas previously targeted for professional 
development. Accountability procedures should be part of a larger professional 
development plan. Collecting data without a purpose is a frivolous practice. School 
psychologists need to consider their professional strengths and weaknesses, identify areas 
for further development, create activities to enhance skill development, and collect 
accountability data to measure progress toward goals. 

Data collection techniques also need to be fairly easy to implement. Elaborate 
research designs are often not practical for the school setting. Surveys should be 
relatively short and focused on specific targeted areas. Outcome data should be easy to 

22 
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collect and incorporated into the intervention plan if possible (e.g., self-monitoring forms, 
behavior checksheets, homework assignment books). 

Finally, data collected should be used to evaluate performance and plan for future 
professional development. Data should be shared with consumers of psychological 
services, supervisors, and other stakeholders. Accountability data can be used to plan and 
evaluate school and system programs and suggest possible areas for training. 
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